Snippet 1

Load in the data set from disk.

using DataFrames
df = readtable("data/01_heights_weights_genders.csv")

	Gender	Height	Weight
1	Male	73.847017017515	241.893563180437
2	Male	68.7819040458903	162.3104725213
3	Male	74.1101053917849	212.7408555565
4	Male	71.7309784033377	220.042470303077
5	Male	69.8817958611153	206.349800623871
6	Male	67.2530156878065	152.212155757083
7	Male	68.7850812516616	183.927888604031
8	Male	68.3485155115879	167.971110489509
9	Male	67.018949662883	175.92944039571
10	Male	63.4564939783664	156.399676387112
11	Male	71.1953822829745	186.604925560358
12	Male	71.6408051192206	213.741169489411
13	Male	64.7663291334055	167.127461073476
14	Male	69.2830700967204	189.446181386738
15	Male	69.2437322298112	186.434168021239
16	Male	67.6456197004212	172.186930058117
17	Male	72.4183166259878	196.028506330482
18	Male	63.974325721061	172.88347020878
19	Male	69.6400598997523	185.98395757313
20	Male	67.9360048540095	182.426648013226
21	Male	67.9150501938206	174.115929081393
22	Male	69.4394398680395	197.73142161472
23	Male	66.1491319608781	149.173566007975
24	Male	75.2059736142212	228.761780615196
25	Male	67.8931963386043	162.006651848287
26	Male	68.1440327982008	192.343976579187
27	Male	69.0896314289256	184.435174408406
28	Male	72.8008435165003	206.828189420354
29	Male	67.4212422817167	175.213922399227
30	Male	68.4964153568827	154.342638925955
⋮	⋮	⋮	⋮

Create a numeric DataArray containing just the heights data.

heights = df[:Height]

10000-element DataArrays.DataArray{Float64,1}:
 73.847 
 68.7819
 74.1101
 71.731 
 69.8818
 67.253 
 68.7851
 68.3485
 67.0189
 63.4565
 71.1954
 71.6408
 64.7663
  ⋮     
 59.5387
 60.9551
 63.1795
 62.6367
 62.0778
 60.0304
 59.0983
 66.1727
 67.0672
 63.868 
 69.0342
 61.9442

median(heights)

66.31807008178464

mean(heights)

66.36755975482124

Snippet 2

Define our own mean and median functions.

function mymean(x) 
    return (sum(x) / length(x))
end

mymean (generic function with 1 method)

function mymedian(x)
    sorted = sort(x)
    if length(x) % 2 == 0
        indices = Array{Int64}(2) 
        indices = [Int64(length(x)/2), Int64(length(x)/2 + 1)]
        return (sorted[indices[1]] + sorted[indices[2]]) / 2
    else 
        index = ceil(length(x)/2)
        return mean(sorted[index])
    end
end

mymedian (generic function with 1 method)

Snippet 3

Compare means and medians on toy examples.

array = [0,100]

2-element Array{Int64,1}:
   0
 100

mean(array)

50.0

median(array)

50.0

array = [0, 0, 100]

3-element Array{Int64,1}:
   0
   0
 100

mean(array)

33.333333333333336

median(array)

0.0

Snippet 4

Confirm that our mean and median functions produce the correct answer.

mymean(heights)

66.36755975482124

mymedian(heights)

66.31807008178464

mean(heights) - mymean(heights)

0.0

median(heights) - mymedian(heights)

0.0

Snippet 5

Experiment with functions for assessing the range of a data set.

minimum(heights)

54.2631333250971

Snippet 6

maximum(heights)

78.9987423463896

Snippet 7

extrema(heights)

(54.2631333250971,78.9987423463896)

Snippet 8

quantile(heights, [0, 0.25, 0.5, 0.75, 1])

5-element Array{Float64,1}:
 54.2631
 63.5056
 66.3181
 69.1743
 78.9987

Snippet 9

quantile(heights, collect(0:0.2:1))

6-element Array{Float64,1}:
 54.2631
 62.859 
 65.1942
 67.4354
 69.8116
 78.9987

Snippet 10

collect(0:0.2:1)

6-element Array{Float64,1}:
 0.0
 0.2
 0.4
 0.6
 0.8
 1.0

Snippet 11

Define a variance function to assess the spread of data.

function myvar(x)
    m = mean(x)
    return sum(map(val -> (val - m) ^ 2, x)) / length(x)
end

myvar (generic function with 1 method)

Snippet 12

Test our variance function for correctness.

myvar(heights)

14.801992292876758

var(heights)

14.803472640140773

Snippet 13

Update the variance function to make it unbiased.

function myunbiasedvar(x)
    m = mean(x)
    return sum(map(val -> (val - m) ^ 2, x)) / (length(x) - 1)
end

myunbiasedvar (generic function with 1 method)

Test our variance function again for correctness.

myunbiasedvar(heights)

14.803472640140772

var(heights)

14.803472640140773

Snippet 14

Check the range predicted by the variance function.

mean(heights) - var(heights), mean(heights) + var(heights)

(51.56408711468047,81.17103239496201)

Snippet 15

mean(heights) - var(heights), mean(heights) + var(heights)

(51.56408711468047,81.17103239496201)

extrema(heights)

(54.2631333250971,78.9987423463896)

Snippet 16

Switch to standard deviations instead for thinking about ranges.

function mystd(x)
    return sqrt(myunbiasedvar(x))
end

mystd (generic function with 1 method)

Snippet 17

Test our standard deviation function for correctness.

mystd(heights)

3.8475281207732284

std(heights)

3.847528120773229

Snippet 18

mean(heights) - std(heights), mean(heights) + std(heights)

(62.52003163404802,70.21508787559448)

extrema(heights)

(54.2631333250971,78.9987423463896)

Snippet 19

mean(heights) - std(heights), mean(heights) + std(heights)

(62.52003163404802,70.21508787559448)

quantile(heights, 0.25), quantile(heights, 0.75)

(63.505620481218955,69.1742617268347)

Snippet 20

Start visualizing data using the ggplot2 package.

using Gadfly
plot(df, x="Height", Geom.histogram(bincount=25))

Snippet 21

plot(df, x="Height", Geom.histogram(bincount=5))

Snippet 22

plot(df, x="Height", Geom.histogram(bincount=10000))

Snippet 23

Experiment with kernel density estimates.

plot(df, x="Height", Geom.histogram)

Snippet 24

Separate out heights and weights based on gender.

plot(df, x="Height", y="Gender", Geom.density)

plot(df, x="Height", y="Gender", color="Gender", Geom.density)

Snippet 25

plot(df, x="Weight", y="Gender", color="Gender", Geom.density)

Snippet 26

Produce two facets in a single plot to make it easier to see the hidden structure.

plot(df, ygroup="Gender" ,x="Weight", y="Gender", color="Gender", Geom.subplot_grid(Geom.density))

Snippet 27

Experiment with random numbers from the normal distribution.

using Distributions

plot(DataFrame(X = rand(Normal(0, 1), 10000) + rand_normal(0, 1)), x="X", Geom.density)

LoadError: UndefVarError: rand_normal not defined
while loading In[47], in expression starting on line 3

Snippet 28

Compare the normal distribution with the Cauchy distribution.

srand(1)
normal_values = rand(Normal(0, 1), 250)
cauchy_values = rand(Cauchy(0, 1), 250)
extrema(normal_values)

(-2.748600587280482,2.2950878238373105)

extrema(cauchy_values)

(-148.8706482930875,81.40805157069148)

Snippet 29

plot(DataFrame(X = normal_values), x="X", Geom.density, Guide.title("Normal distribution"))

plot(DataFrame(X = cauchy_values), x="X", Geom.density, Guide.title("Cauchy distribution"))

Snippet 30

Experiment with random numbers from the gamma distribution.

gamma_values = rand(Gamma(1, 0.001), 100000)
plot(DataFrame(X = gamma_values), x="X", Geom.density, Guide.title("Gamma values"))

Snippet 31

Generate scatterplots of the heights and weights to see their relationship.

plot(df, x="Height", y="Weight", Geom.point)

Snippet 32

Add a smooth shape that relates the two explicitly.

xmin, xmax = extrema(df[:Height])
ymin, ymax = extrema(df[:Weight])
plot(layer(df, x="Height", y="Weight", Geom.point, order = 1),
layer(x=[xmin, xmax],y=[ymin, ymax], Geom.point,Geom.line, Theme(default_color=colorant"purple"), order = 10))

Snippet 33

See how the smooth shape gets better with more data.

• 20 points

plot(df[1:21,:], x="Height", y="Weight", Geom.point)

• 200 points

plot(df[1:201,:], x="Height", y="Weight", Geom.point)

• 2000 points

plot(df[1:2001,:], x="Height", y="Weight", Geom.point)

Snippet 34

Visualize how gender depends on height and weight.

plot(df, x="Height", y="Weight", Geom.point, color="Gender")

Snippet 35

xmin, xmax = extrema(df[:Height])
ymin, ymax = extrema(df[:Weight])
plot(layer(df, x="Height", y="Weight", Geom.point, color="Gender", order = 1),
layer(x=[xmin, xmax],y=[ymin, ymax], Geom.point,Geom.line, Theme(default_color=colorant"purple"), order = 10))